Presenter: HMW Category: graphical models Preference: Oral Evaluation Methods for Topic Models

Statistical topic modeling has become a popular tool for analyzing large, unstructured text collections. There is a significant body of work developing sophisticated topic models and their applications. To date, however, the task of evaluating topic models has not been specifically addressed. Evaluation is an important issue: the unsupervised nature of topic models makes model selection difficult. For some applications there may be extrinsic tasks, such as information retrieval or document classification, for which performance can be evaluated. More universally, however, a topic model’s ability to generalize can be measured by computing the probability of held-out documents under the model, which is independent of any specific application. Here, we consider the simplest topic model, latent Dirichlet allocation (LDA) [? ], and compare a number of methods for estimating the probability of held-out documents given a trained model. Our empirical results on synthetic and real-world data sets show that the estimators currently used in the topic modeling literature are much less accurate and have higher variance than two proposed alternative methods. These proposed alternatives are also applicable to more complicated topic models.